Willow Tech. Report 02-07 Dense Patch Models for Motion Capture from Synchronized Video Streams

This paper proposes a novel approach to nonrigid, markerless motion capture from synchronized video streams acquired by static calibrated cameras. Specifically, we address the problem of estimating the instantaneous geometry of the observed scene as a dense collection of small surface patches, and its nonrigid motion as the velocity field associated with these patches. Starting from a multi-view stereo reconstruction in the first frame, we propose to implement motion capture as a tracking procedure, where the tracks followed by individual patches are expanded both spatially and temporally to nearby patches and frames. The expansion procedure itself is divided into prediction, refinement, and validation steps that effectively exploit both the spatio-temporal consistency of surface appearance and the natural decomposition of nonrigid motion into tangential and normal components. The proposed approach has been implemented, and it is demonstrated on several challenging real data sets.